Recognition device, moving object, recognition method, and storage medium

ABSTRACT

The present invention provides a recognition device that is mounted on a moving object and recognizes a lighting situation of a traffic signal, the recognition device comprising: an imaging unit that periodically images an external environment of the moving object; at least one processor with a memory comprising instructions cause the at least one processor to at least: sequentially detect, for each image periodically obtained by the imaging unit, a lighting mode of the traffic signal included in the image; and determine, in a case where a same lighting mode of the traffic signal is continuously detected for a predetermined time, the lighting mode as a lighting situation of the traffic signal, wherein the predetermined time is twice or more as long as an imaging cycle of the imaging unit.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to and the benefit of Japanese PatentApplication No. 2021-024376 filed on Feb. 18, 2021, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a recognition device, a moving object,a recognition method, and a storage medium.

Description of the Related Art

Japanese Patent Laid-Open No. 2009-61871 discloses a technique in whichrecognition results of signal recognition by a camera and signalrecognition acquired by wireless communication are switched therebetweendepending on a vehicle speed, and either of the recognition results isdisplayed on a display in a vehicle.

In a device that recognizes a lighting situation of a traffic signalbased on an image obtained by an imaging unit (camera), erroneousrecognition may occur due to a difference between an imaging cycle ofthe imaging unit and a lighting cycle of the traffic signal, or thelike. Therefore, it is desirable to improve reliability when recognizingthe lighting situation of the traffic signal using the imaging unit.

SUMMARY OF THE INVENTION

The present invention provides a technology capable of improvingreliability of a lighting situation of a traffic signal recognized byusing an imaging unit, for example.

According to one aspect of the present invention, there is provided arecognition device that is mounted on a moving object and recognizes alighting situation of a traffic signal, the recognition devicecomprising: an imaging unit that periodically images an externalenvironment of the moving object; at least one processor with a memorycomprising instructions, that when executed by the at least oneprocessor, cause the at least one processor to at least: sequentiallydetect, for each image periodically obtained by the imaging unit, alighting mode of the traffic signal included in the image; anddetermine, in a case where a same lighting mode of the traffic signal iscontinuously detected for a predetermined time, the lighting mode as alighting situation of the traffic signal, wherein the predetermined timeis twice or more as long as an imaging cycle of the imaging unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a control device of a vehicle;

FIG. 2 is a block diagram illustrating an example configuration of arecognition device;

FIGS. 3A and 3B are schematic diagrams for explaining recognitionprocessing;

FIGS. 4A and 4B are diagrams for explaining erroneous detection andnon-detection by a detection unit;

FIG. 5 is a diagram schematically illustrating a road on which thevehicle travels;

FIG. 6 is a diagram illustrating an example of external informationsupplied from an external communication apparatus;

FIGS. 7A to 7C are schematic diagrams for explaining recognitionprocessing; and

FIG. 8 is a flowchart illustrating the recognition processing.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note that the following embodiments are notintended to limit the scope of the claimed invention, and limitation isnot made an invention that requires all combinations of featuresdescribed in the embodiments. Two or more of the multiple featuresdescribed in the embodiments may be combined as appropriate.Furthermore, the same reference numerals are given to the same orsimilar configurations, and redundant description thereof is omitted.

First Embodiment

Hereinafter, a recognition device of a first embodiment according to thepresent invention will be described. A recognition device according tothe present invention is a device that is mounted on a moving object andrecognizes a lighting situation (lighting state) of a traffic signalusing an imaging unit (camera) that images an external environment ofthe moving object. Examples of the moving object may include a vehicletraveling on a road. Hereinafter, an example will be described in whichthe recognition device according to the present invention is applied to(mounted on) a four-wheeled vehicle as the vehicle, but the recognitiondevice according to the present invention can also be applied to avehicle other than a four-wheeled vehicle such as a straddle typevehicle (motorcycle, tricycle).

First, an overall control system of a vehicle 1 will be described withreference to FIG. 1. FIG. 1 is a block diagram illustrating a controldevice 2 of the vehicle 1. In FIG. 1, an outline of the vehicle 1 isillustrated in a plan view and a side view, and a sedan-typefour-wheeled passenger vehicle is exemplified as the vehicle 1.

The control device 2 controls each component of the vehicle 1. Thecontrol device 2 includes a plurality of electronic control units (ECUs)20 to 29 communicably connected by an in-vehicle network. Each ECUincludes a processor represented by a CPU, a storage device such as asemiconductor memory, an interface with an external device, and thelike. The storage device stores programs executed by the processor, dataused for processing performed by the processor, and the like. Each ECUmay include a plurality of processors, storage devices, interfaces, andthe like. For example, the ECU 20 includes a processor 20 a and a memory20 b. Processing by the ECU 20 is executed by the processor 20 aexecuting a command included in the program stored in the memory 20 b.Alternatively, the ECU 20 may include a dedicated integrated circuitsuch as an application-specific integrated circuit (ASIC) for executingprocessing by the ECU 20. The same applies to other ECUs.

Hereinafter, functions and the like assigned to each of the ECUs 20 to29 will be described. Note that the number of ECUs and the functionsassigned to the ECUs can be designed as appropriate, and can besubdivided or integrated, as compared with the present embodiment.

The ECU 20 executes vehicle control related to automated driving of thevehicle 1 (self-vehicle) according to the present embodiment. Inautomated driving, at least one of the steering and a vehicle speed(acceleration and deceleration) of the vehicle 1 is automaticallycontrolled. In the present embodiment, an example will be described inwhich both the steering and the vehicle speed of the vehicle 1 areautomatically controlled.

The ECU 21 controls an electric power steering device 3. The electricpower steering device 3 includes a mechanism that steers front wheels inaccordance with a driver's driving operation (steering operation) on asteering wheel 31. In addition, the electric power steering device 3includes a motor that exerts a driving force for assisting the steeringoperation or automatically steering the front wheels, a sensor thatdetects a steering angle, and the like. When the driving state of thevehicle 1 is automated driving, the ECU 21 automatically controls theelectric power steering device 3 in response to an instruction from theECU 20 and controls the traveling direction of the vehicle 1.

The ECUs 22 and 23 control detection units 41 to 43 that detect thesurrounding situation of the vehicle, and perform information processingon detection results. The detection units 41 are each an imaging devicethat periodically images an external environment of the vehicle 1 byimaging (hereinafter, may be referred to as cameras 41). In the presentembodiment, the cameras 41 are attached to a vehicle interior side ofthe windshield at a front portion of the roof of the vehicle 1 so as tobe able to capture an image of ahead of the vehicle 1. By analyzing(performing image processing of) the images captured by the cameras 41,it is possible to analyze a target such as a traffic signal locatedahead of the vehicle 1, and to extract display of the traffic signal anda division line (white line or the like) of a lane on a road.

The detection units 42 (lidar detection units) are each a lightdetection and ranging (LIDAR) (hereinafter, may be referred to as lidars42), and detect a target around the vehicle 1 and measure a distance tothe target by light. In the present embodiment, five lidars 42 areprovided, one at each corner portion of a front portion of the vehicle1, one at the center of a rear portion of the vehicle 1, and one at eachside of the rear portion of the vehicle 1. The detection units 43 (radardetection units) are each a millimeter-wave radar (hereinafter, may bereferred to as radars 43), and detect a target around the vehicle 1 andmeasure a distance to the target by radio waves. In the presentembodiment, five radars 43 are provided, one at the center of the frontportion of the vehicle 1, one at each corner portion of the frontportion of the vehicle 1, and one at each corner portion of the rearportion of the vehicle 1.

The ECU 22 controls one camera 41 and each lidar 42 and performsinformation processing on the detection results. The ECU 23 controls theother camera 41 and each radar 43 and performs information processing onthe detection results. Since two sets of devices for detecting thesurrounding situation of the vehicle are provided, the reliability ofthe detection results can be improved, and since different types ofdetection units such as a camera, a lidar, and a radar are provided, thesurrounding environment of the vehicle can be analyzed in multiple ways.In the present embodiment, each of the ECU 22 and the ECU 23 canfunction as a detection unit that sequentially detects, for each imageperiodically obtained by the camera 41, a lighting mode of a trafficsignal included in the image. Here, the display of the traffic signalmay include color display (lighting colors (red, green, and yellow)) ofthe traffic signal for indicating whether traveling of the vehicle ispermitted, and arrow display of the traffic signal for indicating atravelable direction of the vehicle.

The ECU 24 controls a gyro sensor 5, a global positioning system (GPS)sensor 24 b, and a communication device 24 c, and performs informationprocessing of detection results or a communication result. The gyrosensor 5 detects a rotational motion of the vehicle 1. The course of thevehicle 1 can be determined based on the detection result of the gyrosensor 5, the wheel speed, and the like. The GPS sensor 24 b detects thecurrent location of the vehicle 1. The communication device 24 cperforms wireless communication with a server that provides mapinformation and traffic information, and acquires these pieces ofinformation. The communication device 24 c performs wirelesscommunication, for example, with an information providing server of asystem for monitoring a traffic condition of a road such as VehicleInformation and Communication System (VICS) (registered trademark), andacquires traffic information indicating a traffic condition of a road onwhich a self-vehicle M is traveling or a road on which the self-vehicleM is scheduled to travel. The traffic information includes the followingtypes of information regarding a set route: traffic jam information,information regarding time required to pass through a traffic jam point,and traffic hazard information such as information regarding abroken-down vehicle/road construction, information regarding speedregulation/lane regulation, and the like.

The ECU 24 can access a database 24 a of map information constructed inthe storage device, and the ECU 24 searches for a route from the currentlocation to a destination, and the like. The database 24 a can bearranged on a network, and the communication device 24 c can access thedatabase 24 a on the network to acquire information.

The ECU 25 includes a communication device 25 a capable of performingvehicle-to-vehicle communication, road-to-vehicle communication, orcommunication with an information processing device such as asmartphone. For example, the communication device 25 a can performwireless communication with other surrounding vehicles, exchangeinformation between the vehicles, or exchange information by wirelesscommunication with an external information processing device or thelike. In the present embodiment, the communication device 25 a and thecommunication device 24 c can each function as a reception unit thatreceives information indicating a lighting situation of a traffic signal(lighting information regarding color display and arrow display) bywireless communication.

The ECU 26 controls a power plant 6. The power plant 6 is a mechanismthat outputs a driving force for rotating driving wheels of the vehicle1 and includes, for example, an engine and a transmission. Note that theconfiguration of the power plant 6 is not limited to this example, andincludes an electric vehicle using an electric motor as a power source,a hybrid vehicle including an engine and an electric motor, and thelike. Among them, the electric vehicle is driven using electric powerdischarged by a battery such as a secondary battery, a hydrogen fuelcell, a metal fuel cell, or an alcohol fuel cell, for example.

For example, the ECU 26 controls the output of the engine in response toa driving operation (accelerator operation or acceleration operation) ofthe driver detected by an operation detection sensor 7 a provided on anaccelerator pedal 7A, and switches the gear ratio of the transmissionbased on information regarding a vehicle speed detected by a vehiclespeed sensor 7 c and the like. When the driving state of the vehicle 1is automated driving, the ECU 26 automatically controls the power plant6 in response to an instruction from the ECU 20 and controls the vehiclespeed (speed, acceleration and deceleration of the vehicle 1).

The ECU 27 controls lighting devices (headlights, taillights, and thelike) including direction indicators 8 (blinkers). In the example ofFIG. 1, the direction indicators 8 are provided at the front portion,the door mirrors, and the rear portion of the vehicle 1.

The ECU 28 controls an input/output device 9. The input/output device 9outputs information to occupants including the driver and receives aninput of information from the occupants. A sound output device 91notifies the occupants of information by sound. A display device 92notifies the driver of information by displaying an image. The displaydevice 92 is arranged in front of the driver seat and the passengerseat, for example, and constitutes a touch panel type instrument panelor the like functioning as a human-machine interface.

For example, on a route from the current location to a destinationsearched for by the ECU 24, the ECU 28 performs display control todisplay, on the display device 92, road information acquired fromlocation information regarding the vehicle 1 (self-vehicle) and mapinformation, and including information regarding a plurality of lanes onwhich the vehicle 1 is currently traveling. Furthermore, the ECU 28controls the sound output device 91 and the display device 92 to providethe driver with sound, map display, and guidance information by sound.

Note that, although the sound and the display have been exemplifiedhere, information notification may also be made by using vibration orlight. In addition, information notification may be made by using acombination of some of sound, display, vibration, and light.Furthermore, the combination or the notification mode may vary dependingon the level (for example, the degree of urgency) of information thatshould be notified.

An input device 93 is arranged at a position operable by the driver, andincludes a switch group for inputting an instruction to the vehicle 1and a sound input device to which a voice of an occupant is input.

The ECU 29 controls a brake device 10 and a parking brake (notillustrated). The brake device 10 is, for example, a disc brake device,and is provided on each wheel of the vehicle 1 to decelerate or stop thevehicle 1 by applying resistance to the rotation of the wheel. The ECU29 controls the operation of the brake device 10 in response to adriving operation (brake operation) of the driver detected by anoperation detection sensor 7 b provided on a brake pedal 7B, forexample. When the driving state of the vehicle 1 is automated driving,the ECU 29 automatically controls the brake device 10 in response to aninstruction from the ECU 20 and controls the deceleration and stop ofthe vehicle 1. The brake device 10 and the parking brake can also beoperated to maintain a stopped state of the vehicle 1. In addition, in acase where the transmission of the power plant 6 includes a parking lockmechanism, the parking lock mechanism can also be operated to maintainthe stopped state of the vehicle 1.

[Configuration of Recognition Device]

Next, an example configuration of a recognition device 100 of thepresent embodiment will be described with reference to FIG. 2. FIG. 2 isa block diagram illustrating an example configuration of the recognitiondevice 100 of the present embodiment. As described above, therecognition device 100 is a device that recognizes a lighting situation(lighting state) of a traffic signal using a camera, and may include animaging unit (image capturing unit) 110, a processing unit 120, adisplay unit 130, and a communication unit 140 connected to becommunicable with one another via a system bus 150.

The imaging unit 110 is, for example, the camera 41 in FIG. 1, andperiodically images the external environment of the vehicle 1 (i.e.captures an image of the external environment of the vehicle 1) at apredetermined imaging cycle (for example, 60 msec). The imaging unit 110may be understood as an external environment sensor that acquiresexternal environment information regarding a target including an objector a sign present around the vehicle 1. Note that, hereinafter, theimaging unit 110 may be referred to as a camera 110.

The processing unit 120 is, for example, the control device 2 (ECUs 20to 29) in FIG. 1, and can be constituted with a computer including aprocessor represented by a CPU, a storage device such as a semiconductormemory, an interface with an external device, and the like. In thepresent embodiment, a program (hereinafter, may be referred to as arecognition program) for recognizing a lighting state of a trafficsignal is stored in the storage device, and the processor executes acommand included in the recognition program, and thereby recognitionprocessing of the lighting situation of the traffic signal can beexecuted by the processing unit 120. The processing unit 120 of thepresent embodiment may include a detection unit 121, a determinationunit 122 (approval unit), and an output unit 123.

The detection unit 121 is, for example, the ECU 23 and the ECU 24 inFIG. 1, and sequentially detects, for each image periodically obtainedby the camera 110, a lighting mode of a traffic signal included in theimage. Specifically, by performing known image processing on the imageobtained by the camera 110, the detection unit 121 can specify (extract)a traffic signal as a target (object) included in the image and candetect a lighting mode of the specified traffic signal. The lightingmode of the traffic signal may include, for example, what color of lightof color display (green, yellow, and red) of the traffic signal isturned on, whether arrow display of the traffic signal is turned on, andthe like.

The determination unit 122 is, for example, the ECU 20 in FIG. 1, anddetermines (approves, recognizes) the lighting situation of the trafficsignal based on the lighting mode of the traffic signal detected by thedetection unit 121. In the present embodiment, when the same lightingmode of the traffic signal is continuously detected for a predeterminedtime by the detection unit 121, the determination unit 122 determines(approves, recognizes) the lighting mode continuously detected for thepredetermined time as a current (actual) lighting situation of thetraffic signal. The predetermined time can be set to be twice or more aslong as an imaging cycle of the camera 110. The predetermined time maybe defined by a predetermined number of times the same lighting mode ofthe traffic signal is continuously detected by the detection unit 121with respect to the images periodically obtained by the camera 110. Inother words, the predetermined number of times is the number of images(that is, the number of times of imaging) in which the same lightingmode is continuously detected by the detection unit 121, and can be setto be two or more. Hereinafter, an example will be described in whichthe determination unit 122 makes a determination using a predeterminednumber of times, and the predetermined number of times is set to bethree.

The output unit 123 is, for example, the ECU 28 in FIG. 1, and outputsinformation (hereinafter, may be referred to as lighting information)indicating the lighting situation of the traffic signal determined bythe determination unit 122 to the display unit 130, and causes thedisplay unit 130 to display the lighting information. The output unit123 may output the lighting information obtained from the determinationunit 122 to a unit in charge of automated driving control in the ECU 20so that the lighting information is used in automated driving (vehiclecontrol) of the vehicle 1 executed by the ECU 20.

The display unit 130 is, for example, a display corresponding to thedisplay device 92 in FIG. 1, and displays the lighting informationoutput from the output unit 123. The communication unit 140 is, forexample, the communication device 24 c and the communication device 25 ain FIG. 1, and receives, by wireless communication, informationindicating a lighting situation of a traffic signal from an externalcommunication apparatus (transmission apparatus) installed on a road onwhich the vehicle 1 travels. The external communication apparatus can beconfigured as, for example, a communication apparatus of a beacon systemor a cellular system using a mobile communication network standardizedby a communication standard for mobile phones. The communication unit140 can acquire information indicating lighting situations of aplurality of traffic signals present around the vehicle 1 (as anexample, a plurality of traffic signals arranged in the travelingdirection of the vehicle 1) from the external communication apparatus.Hereinafter, the information received from the external communicationapparatus may be referred to as external information.

[Outline of Processing]

Next, an outline of recognition processing executed by the processingunit 120 will be described. As described above, regarding the lightingmode of the traffic signal detected by the detection unit 121 from theimages periodically obtained by the camera 110, when the same lightingmode is continuously detected a predetermined number of times (here,three times), the processing unit 120 (determination unit 122)determines the continuously detected lighting state as the current(actual) lighting situation of the traffic signal.

FIGS. 3A and 3B are schematic diagrams for explaining recognitionprocessing for recognizing a lighting situation of a traffic signal TSin the processing unit 120 (determination unit 122). FIG. 3Aillustrates, in time series, information input to the determination unit122, that is, information indicating a lighting mode of the trafficsignal TS detected by the detection unit 121. Specifically, FIG. 3Aillustrates, in time series, results of detecting the lighting mode ofthe traffic signal TS included in images obtained by the camera 110 byperforming image processing on the images in the detection unit 121.FIG. 3B illustrates, in time series, information output from thedetermination unit 122, that is, information (lighting information)indicating the lighting situation of the traffic signal TS determined bythe determination unit 122. Specifically, FIG. 3B illustrates, in timeseries, lighting information regarding the traffic signal TS determinedby the determination unit 122 based on the lighting mode of the trafficsignal TS detected by the detection unit 121. Here, “t” in FIGS. 3A and3B represents a time series number of an image periodically obtained bythe camera 110, that is, a time series number of information indicatingthe lighting mode of the traffic signal TS detected by the detectionunit 121 from each image, but may be understood as time corresponding toan imaging cycle of the camera 110 (in other words, a detection cycle ofthe detection unit 121). “t=1, 2, 3, or 4” is only used to represent, intime series, a part of the recognition processing continuously performedby the processing unit 120 extracted therefrom and does not representthat the part is obtained first, second, third, or fourth in therecognition processing. The same applies to “t” in FIGS. 4A and 4B andFIGS. 7A to 7C described later.

When t=1, the detection unit 121 detects that the lighting mode of thetraffic signal TS is red lighting R from an image obtained by the camera110, and the determination unit 122 also determines that the lightingsituation of the traffic signal TS is red lighting R. Next, when t=2,the detection unit 121 detects that the lighting mode of the trafficsignal TS is green lighting B from an image obtained next by the camera110 as illustrated in FIG. 3A. However, at that stage, since the numberof times the lighting mode (green lighting B) of the traffic signal TSis continuously detected by the detection unit 121 is still one and hasnot reached the predetermined number of times (three times), thedetermination unit 122 holds the lighting situation (red lighting R) ofthe traffic signal TS previously determined as illustrated in FIG. 3B.

When t=3, the detection unit 121 detects that the lighting mode of thetraffic signal TS is green lighting B from an image obtained next by thecamera 110 as illustrated in FIG. 3A. However, at that stage, since thenumber of times the lighting mode (green lighting B) of the trafficsignal TS is continuously detected by the detection unit 121 is stilltwo and has not reached the predetermined number of times (three times),the determination unit 122 holds the lighting situation (red lighting R)of the traffic signal TS previously determined as illustrated in FIG.3B. Next, when t=4, the detection unit 121 detects that the lightingmode of the traffic signal TS is green lighting B from an image obtainednext by the camera 110 as illustrated in FIG. 3A. At that stage, sincethe number of times the lighting mode (green lighting B) of the trafficsignal TS is continuously detected by the detection unit 121 reaches thepredetermined number of times (three times), the determination unit 122determines that the lighting situation of the traffic signal TS is greenlighting B as illustrated in FIG. 3B.

By executing the above recognition processing, the determination unit122 of the present embodiment can accurately determine (recognize) thelighting situation of the traffic signal TS even when there occurserroneous detection and/or non-detection by the detection unit 121. Thatis, it is possible to avoid or reduce determination of the lightingsituation of the traffic signal TS based on the erroneous detectionand/or non-detection by the detection unit 121. What is meant by theerroneous detection by the detection unit 121 is that, as indicated byt=2 in FIG. 4A, although the red light of the traffic signal TS isactually turned on, the detection unit 121 detects green lighting B.What is meant by the non-detection by the detection unit 121 is that, asindicated by t=2 in FIG. 4B, although the (red) light of the trafficsignal TS is actually turned on, the detection unit 121 does not detectthe lighting of the traffic signal TS. In addition, the information(lighting information) indicating the lighting situation of the trafficsignal TS determined by the determination unit 122 by the recognitionprocessing described above can be supplied to the display unit 130(display) by the output unit 123 and displayed on the display unit 130.As described above, the lighting information output from thedetermination unit 122 may be used for automated driving (vehiclecontrol) of the vehicle 1 executed by the ECU 20.

Here, when external information indicating a lighting situation of atraffic signal is received from the external communication apparatus bythe communication unit 140, the determination unit 122 of the presentembodiment may determine the lighting situation of the traffic signal TSbased on the external information. Such external information obtainedfrom the external communication apparatus tends to be more reliable asinformation indicating a lighting situation of a traffic signal.Therefore, even if a time (the number of times) when the same lightingmode of the traffic signal TS is continuously detected by the detectionunit 121 is less than a predetermined time (predetermined number oftimes), when the communication unit 140 receives external informationfrom the external communication apparatus, the determination unit 122can determine the lighting situation of the traffic signal based on theexternal information.

FIG. 5 schematically illustrates a road on which the vehicle 1 travels.In the example illustrated in FIG. 5, the traveling direction of thevehicle 1 is the direction of arrow A, and a plurality of trafficsignals TS1 to TS3 are arranged in the traveling direction of thevehicle 1. Each of the traffic signals TS1 to TS3 includes color display(lighting colors (red, green, and yellow)) for indicating whethertraveling is permitted, straight arrow display for permitting travelingstraight, and right turn arrow display for permitting turning right. Inaddition, a plurality of external communication apparatuses CA1 and CA2are arranged on the road. The external communication apparatus CA1 isarranged in front of the traffic signal TS1, and the externalcommunication apparatus CA2 is arranged in front of the traffic signalTS2 (between the traffic signal TS1 and the traffic signal TS2). Each ofthe external communication apparatuses CA1 and CA2 is an apparatus thattransmits (provides), to the vehicle 1 by a beacon system, externalinformation indicating lighting situations of a plurality of trafficsignals (including traffic signals TS1 to TS3) present around thevehicle 1. As an example, the external communication apparatus CA1arranged in front of the traffic signal TS1 supplies (transmits)external information indicating the lighting situation of each of thetraffic signals TS1 to TS3 to the vehicle 1 as illustrated in FIG. 6.FIG. 6 illustrates an example of the external information supplied fromthe external communication apparatus CA1.

FIGS. 7A to 7C are schematic diagrams for explaining recognitionprocessing in the processing unit 120 (determination unit 122). FIGS. 7Ato 7C each illustrate an example of recognition processing forrecognizing the lighting situation of the traffic signal TS based notonly on the lighting mode of the traffic signal TS detected by thedetection unit 121 but also on the external information obtained fromthe external communication apparatus CA. FIG. 7A illustrates, in timeseries, information input to the determination unit 122, that is,information indicating the lighting mode of the traffic signal TSdetected by the detection unit 121. FIG. 7B illustrates, in time series,the external information indicating the lighting mode of the trafficsignal TS supplied from the external communication apparatus (forexample, CA1). FIG. 7C illustrates, in time series, information outputfrom the determination unit 122, that is, lighting informationindicating the lighting situation of the traffic signal TS determined bythe determination unit 122.

When t=1, the detection unit 121 detects that the lighting mode of thetraffic signal TS is red lighting R from an image obtained by the camera110, and the determination unit 122 also determines that the lightingsituation of the traffic signal TS is red lighting R. At that stage, theexternal information is not received yet. Next, when t=2, the detectionunit 121 detects that the lighting mode of the traffic signal TS isgreen lighting B from an image obtained next by the camera 110 asillustrated in FIG. 7A. At that stage, the external information is notreceived yet. In addition, the number of times the lighting mode (greenlighting B) of the traffic signal TS is continuously detected by thedetection unit 121 is still one and has not reached the predeterminednumber of times (three times). Therefore, as illustrated in FIG. 7C, thedetermination unit 122 holds the lighting situation (red lighting R) ofthe traffic signal TS previously determined.

When t=3, the detection unit 121 detects that the lighting mode of thetraffic signal TS is green lighting B from an image obtained next by thecamera 110 as illustrated in FIG. 7A. On the other hand, thecommunication unit 140 receives, from the external communicationapparatus CA1, external information indicating that the lightingsituation of the traffic signal TS is green lighting B. At that stage,the number of times the lighting mode (green lighting B) of the trafficsignal TS is continuously detected by the detection unit 121 is stilltwo and has not reached the predetermined number of times (three times),but the external information indicating green lighting B is obtainedfrom the external communication apparatus as illustrated in FIG. 7B.Therefore, as illustrated in FIG. 7C, the determination unit 122preferentially uses the external information to determine that thelighting situation of the traffic signal TS is green lighting B. Next,when t=4, the lighting mode of the traffic signal TS detected by thedetection unit 121 from an image obtained next by the camera 110 isgreen lighting B (see FIG. 7A), and the external information alsoindicates green lighting B (see FIG. 7B). Therefore, as illustrated inFIG. 7C, the determination unit 122 determines that the lightingsituation of the traffic signal TS is green lighting B. By further usingthe external information supplied from the external communicationapparatus as described above, it is possible to accurately and quicklydetermine (recognize) the lighting situation of the traffic signal TS.

[Flow of Recognition Processing]

Next, a flow of recognition processing executed by the processing unit120 will be described. FIG. 8 is a flowchart illustrating therecognition processing executed by the processing unit 120. Theflowchart illustrated in FIG. 8 is repeatedly executed, and can be newlystarted from step S11 even after step S16 ends.

In step S11, the processing unit 120 causes the camera 110 to image theexternal environment (in the present embodiment, ahead) of the vehicle1, thereby acquiring an image of the external environment of the vehicle1. In step S12, the processing unit 120 (detection unit 121) detects thelighting mode of the traffic signal TS included in the image obtained bythe camera 110. For example, the detection unit 121 can detect, as thelighting mode, the lighting color of the traffic signal TS in the imageby performing known image processing on the image obtained by the camera110. Next, in step S13, the processing unit 120 (determination unit 122)determines whether the same lighting mode has been continuously detecteda predetermined number of times (in the present embodiment, three times)regarding the lighting mode of the traffic signal TS detected by thedetection unit 121. If the same lighting mode has been continuouslydetected the predetermined number of times, the process proceeds to stepS16, and if the same lighting mode has not been continuously detectedthe predetermined number of times, the process proceeds to step S14.

In step S14, the processing unit 120 (determination unit 122) determineswhether the external information has been acquired from the externalcommunication apparatus CA by the communication unit 140. As describedabove, the external information is information indicating the lightingsituation of the traffic signal TS, and can be acquired, for example, bya beacon system or a cellular system. If the external information hasbeen acquired, the process proceeds to step S15, and if the externalinformation has not been acquired, the process returns to step S11.Next, in step S15, the processing unit 120 (determination unit 122)determines whether an elapsed time from the acquisition of the externalinformation by the communication unit 140 in step S14 is within aprescribed period. Since the lighting situation of the traffic signal TSchanges from moment to moment, if the lighting situation of the trafficsignal TS in the acquired external information is out-of-date one, itbecomes difficult to accurately determine the current lighting situationof the traffic signal TS in step S16 described later. Therefore, in thisstep S15, a prescribed period (prescribed time) is provided as a periodof validity in which the external information can be used, and it isdetermined whether the elapsed time from the acquisition of the externalinformation is within the prescribed period. The prescribed period ispreferably set based on information indicating a time during which thesame lighting color of the traffic signal TS remains turned on (forexample, green lighting time and red lighting time). The informationindicating the time during which the same lighting color remains turnedon may be, for example, information regarding time indicated in“scheduled change” in FIG. 6. The prescribed period may be set for eachtraffic signal TS, or may be commonly set for a plurality of trafficsignals TS.

Here, as described above, the external information received from theexternal communication apparatus CA may include information indicatinglighting situations of the plurality of traffic signals TS presentaround the vehicle 1. In that case, the determination unit 122 candetermine the current lighting situation regarding, among the pluralityof traffic signals TS, a traffic signal TS of which the elapsed timefrom the acquisition of the external information is within theprescribed period based on the external information. On the other hand,the current lighting situation regarding, among the plurality of trafficsignals TS, a traffic signal TS of which the elapsed time from theacquisition of the external information is not within the prescribedperiod is preferably determined based on the detection result of thedetection unit 121 without using the external information.

In step S16, the processing unit 120 (determination unit 122) determines(approves, recognizes) the lighting situation of the traffic signal TS.For example, when the same lighting mode of the traffic signal TS iscontinuously detected the predetermined number of times (three times) bythe detection unit 121, the determination unit 122 determines thelighting mode detected by performing measurement the predeterminednumber of times as the current (actual) lighting situation of thetraffic signal TS. On the other hand, when the external information isreceived from the external communication apparatus CA and the elapsedtime from the reception of the external information is within theprescribed period, the determination unit 122 preferentially uses theexternal information to determine the lighting situation of the trafficsignal TS included in the external information as the current (actual)lighting situation of the traffic signal TS.

Here, in the above example, if the lighting state is within theprescribed period as a result of checking the prescribed period in stepS15, the lighting state is determined in step S16, but as anothermethod, the processing of the prescribed period may not be performed instep S15, and traffic signal information in which the lighting statedetected by the detection unit 121 in step S16 matches the externalinformation received from the external communication apparatus CA may bedetermined as the lighting state. For example, in a case where thedetection unit 121 detects the green color of the traffic signal, andthe external information received from the external communicationapparatus CA also indicates that the traffic signal is green, thelighting state of the traffic signal may be determined to be greenwithout considering the prescribed period.

Furthermore, even in a case where the prescribed time is not reflected,when display information regarding the traffic signal is received fromthe external communication apparatus CA, it is possible to utilize thefact that a lighting color and a traveling direction indicated by atraffic signal arranged in the traveling direction, and in additionthereto, a minimum lighting time and a maximum lighting time thereof aretransmitted from the external communication apparatus CA. For example,there is a case of communication including the following information: alighting color is green, the minimum lighting time is zero seconds, andthe maximum lighting time is 10 seconds. In that case, although trafficsignal information is successfully acquired by the communication, thereis a possibility that the lighting color of the traffic signal ofinterest will be changed to yellow after zero seconds, and it ispossible to determine the certainty, for example, to determine that theinformation is uncertain as lighting prediction information regardingthe traffic signal.

As described above, by performing display after comparing the externalinformation (traffic signal information) received by the communicationunit 140 and the information obtained by the imaging unit 110, highlyaccurate display can be performed. At the time of comparison, areference time may be set in consideration of the minimum lighting time.That is, when a minimum reference time (minimum lighting time) is longerthan an estimated time until a moving object enters an intersection, theinformation received by the communication unit 140 (externalinformation) is prioritized, and on the other hand, when the minimumreference time is shorter, the information received by the communicationunit 140 and the information obtained by the detection unit 121 may becompared with each other.

As described above, the recognition device 100 of the present embodimentdetects, for each image periodically obtained by the camera 110, thelighting mode of the traffic signal TS in the image, and when the samelighting mode is continuously detected a predetermined number of times,the recognition device 100 determines the lighting mode as the current(actual) lighting situation of the traffic signal TS. As a result, evenwhen erroneous detection or non-detection occurs at the time ofdetecting the lighting mode of the traffic signal TS from the image, itis possible to avoid or reduce determination (recognition) of thecurrent lighting situation of the traffic signal TS based on theerroneous detection or the non-detection. That is, it is possible toaccurately and precisely recognize the current lighting situation of thetraffic signal TS and to improve the reliability of the lightingsituation of the traffic signal recognized using the imaging unit.

Second Embodiment

In the first embodiment, the example has been described in which theabove recognition processing is performed on the color display of thetraffic signal TS, but the above recognition processing can also beperformed on the arrow display of the traffic signal TS. However, it isgenerally difficult to detect the arrow display by known imageprocessing, and erroneous detection or non-detection by the detectionunit 121 is likely to occur. Therefore, when the recognition processingis performed on the arrow display, instead of the predetermined timeused for the color display, a second predetermined time set to a timelonger than the predetermined time is preferably used. Similarly,instead of the predetermined number of times used for the color display,a second predetermined number of times set to a number of times largerthan the predetermined number of times may be used. Note that theconfiguration of the recognition device 100 and the contents ofprocessing other than those described above are similar to those of thefirst embodiment, and thus the description thereof will be omitted here.

Third Embodiment

As described in the first embodiment, the lighting information(determination result) indicating the lighting situation of the trafficsignal determined by the determination unit 122 may be used forautomated driving by the ECU 20. When the lighting information is usedfor automated driving as described above, the lighting information maybe required to be highly precise. Therefore, when the lightinginformation is used for automated driving, the predetermined time ispreferably set to be longer (alternatively, the predetermined number oftimes is set to be larger) than that when the lighting information isnot used for automated driving. For example, in a case where thepredetermined number of times is set to three when the lightinginformation is used only for display on the display unit 130, thepredetermined number of times is preferably set to four or a valuelarger than that when the lighting information is used for automateddriving. As a result, the lighting information can be obtained moreprecisely, so that the accuracy of the automated driving can be improvedand discomfort to an occupant can be reduced. Note that theconfiguration of the recognition device 100 and the contents ofprocessing other than those described above are similar to those of thefirst embodiment, and thus the description thereof will be omitted here.In the present embodiment, the second embodiment can also be applied.

Fourth Embodiment

The image obtained by the camera 110 may include, in addition to thetraffic signal TS that performs indication regarding traveling in thetraveling direction of the vehicle 1, a plurality of traffic signalsincluding a traffic signal that performs indication regarding travelingin a direction different from the traveling direction of the vehicle 1and a traffic signal at an intersection located further ahead. When theplurality of traffic signals are included in the image obtained by thecamera 110 as described above, the detection unit 121 preferablyobtains, for each traffic signal, the degree of certainty (likelihood,reliability) indicating certainty that it is a traffic signal in thetraveling direction of the vehicle 1, and detects a lighting mode of atraffic signal of which the degree of certainty is highest among theplurality of traffic signals. For example, the detection unit 121 cancalculate the area of a lighting portion in the image for each of theplurality of traffic signals included in the image, and can employ thecalculated size of the area as the degree of certainty. By calculatingthe degree of certainty as described above, it is possible to specifythe traffic signal TS in the traveling direction of the vehicle 1 evenwhen the plurality of traffic signals are included in the image. Thedegree of certainty may be improved by integrating information regardingthe plurality of traffic signals and then collating the integratedinformation with other map information or information obtained bycommunication. Note that the configuration of the recognition device 100and the contents of processing other than those described above aresimilar to those of the first embodiment, and thus the descriptionthereof will be omitted here. In the present embodiment, the second andthird embodiments can also be applied.

Summary of Embodiments

1. The recognition device according to the above embodiment is:

a recognition device (for example, 100) that is mounted on a movingobject (for example, 1) and recognizes a lighting situation of a trafficsignal (for example, TS), the recognition device including:

an imaging unit (for example, 110) that periodically images an externalenvironment of the moving object;

a detection unit (for example, 121) that sequentially detects, for eachimage periodically obtained by the imaging unit, a lighting mode of thetraffic signal included in the image; and

a determination unit (for example, 122) that, when a same lighting modeof the traffic signal is continuously detected for a predetermined timeby the detection unit, determines the lighting mode as a lightingsituation of the traffic signal, in which

the predetermined time is twice or more as long as an imaging cycle ofthe imaging unit.

According to this embodiment, even when erroneous detection ornon-detection occurs at the time of detecting the lighting mode of thetraffic signal from the image, it is possible to avoid or reducedetermination (recognition) of the current lighting situation of thetraffic signal based on the erroneous detection or the non-detection.That is, it is possible to accurately and precisely recognize thecurrent lighting situation of the traffic signal and to improve thereliability of the lighting situation of the traffic signal recognizedusing the imaging unit.

2. In the above embodiment,

when the same lighting mode of the traffic signal is not continuouslydetected for the predetermined time by the detection unit, thedetermination unit holds a lighting situation of the traffic signalpreviously determined.

According to this embodiment, the information indicating the lightingsituation of the traffic signal is not changed as long as the currentlighting situation of the traffic signal cannot be reliably determined,so that the accuracy can be improved.

3. In the above embodiment,

the predetermined time is defined by a predetermined number of times thesame lighting mode of the traffic signal is continuously detected by thedetection unit, and

the predetermined number of times is two or more.

According to this embodiment, even in a case where the predeterminednumber of times is used instead of the predetermined time, it ispossible to accurately and precisely recognize the current lightingsituation of the traffic signal as in a case where the predeterminedtime is used.

4. In the above embodiment,

further included is an output unit (for example, 123) that outputsinformation indicating the lighting situation of the traffic signaldetermined by the determination unit to a display unit (for example,130).

According to this embodiment, it is possible to notify a user (forexample, a driver) of the lighting situation of the traffic signal.

5. In the above embodiment,

the determination unit performs the determination of the lightingsituation of the traffic signal using the predetermined time on colordisplay of the traffic signal that indicates whether traveling of themoving object is permitted.

According to this embodiment, it is possible to accurately and preciselyrecognize the lighting situation of the color display (lighting colors(red, green, and yellow)) of the traffic signal.

6. In the above embodiment,

when determining a lighting situation of arrow display of the trafficsignal indicating a travelable direction of the moving object, thedetermination unit uses, instead of the predetermined time, a secondpredetermined time longer than the predetermined time.

According to this embodiment, it is possible to accurately and preciselyrecognize the lighting situation even for the arrow display of thetraffic signal which is difficult to detect by known image processing,and of which erroneous detection or non-detection is likely to occur.

7. In the above embodiment,

further included is a communication unit (for example, 140) thatreceives information related to lighting of the traffic signal bywireless communication, and

even when a time during which the same lighting mode of the trafficsignal is continuously detected by the detection unit is less than thepredetermined time, the determination unit determines the lightingsituation of the traffic signal based on the information when thecommunication unit receives the information.

According to this embodiment, information received by wirelesscommunication tends to be more reliable than information detected froman image, so that by actively using such information, it is possible tomore accurately and precisely recognize the lighting situation of thetraffic signal.

8. In the above embodiment,

the communication unit receives, from an external communicationapparatus (for example, CA) installed on a road on which the movingobject travels, the information regarding a plurality of traffic signalspresent around the moving object, and

the determination unit

determines a lighting situation regarding, among the plurality oftraffic signals, a traffic signal of which an elapsed time fromreception of the information by the communication unit is within aprescribed period based on the information, and

determines the lighting situation regarding a traffic signal of whichthe elapsed time is not within the prescribed period based on a resultof detection by the detection unit.

According to this embodiment, it is possible to appropriately recognizethe lighting situation of each traffic signal depending on a situationof information acquisition by the recognition device.

9. In the above embodiment,

when a plurality of traffic signals are included in an image obtained bythe imaging unit, the detection unit obtains a degree of certainty foreach of the traffic signals, the degree of certainty indicatingcertainty that the traffic signal is a traffic signal in the travelingdirection of the moving object, and detects a lighting mode of a trafficsignal of which the degree of certainty is highest among the pluralityof traffic signals.

According to this embodiment, even when the plurality of traffic signalsare included in the image acquired by the imaging unit, it is possibleto specify a traffic signal in the traveling direction of the vehicleand to detect the lighting mode of the specified traffic signal. Thatis, it is possible to recognize the lighting situation of the trafficsignal in the traveling direction of the vehicle.

10. In the above embodiment,

when a determination result of the lighting situation of the trafficsignal is used for automatic control of steering and a vehicle speed ofthe moving object, the determination unit sets the predetermined timelonger than that when the determination result is not used for theautomatic control.

According to this embodiment, when the determination result of thelighting situation of the traffic signal is used for automated driving,the determination result can be obtained more precisely, so that theaccuracy of the automated driving can be improved and discomfort to anoccupant can be reduced.

The invention is not limited to the foregoing embodiments, and variousvariations/changes are possible within the spirit of the invention.

What is claimed is:
 1. A recognition device that is mounted on a moving object and recognizes a lighting situation of a traffic signal, the recognition device comprising: an imaging unit that periodically images an external environment of the moving object; at least one processor with a memory comprising instructions, that when executed by the at least one processor, cause the at least one processor to at least: sequentially detect, for each image periodically obtained by the imaging unit, a lighting mode of the traffic signal included in the image; and determine, in a case where a same lighting mode of the traffic signal is continuously detected for a predetermined time, the lighting mode as a lighting situation of the traffic signal, wherein the predetermined time is twice or more as long as an imaging cycle of the imaging unit.
 2. The recognition device according to claim 1, wherein in a case where the same lighting mode of the traffic signal is not continuously detected for the predetermined time, the at least one processor is configured to hold a lighting situation of the traffic signal previously determined.
 3. The recognition device according to claim 1, wherein the predetermined time is defined by a predetermined number of times the same lighting mode of the traffic signal is continuously detected, and the predetermined number of times is two or more.
 4. The recognition device according to claim 1, wherein the at least one processor is configured to output information indicating the determined lighting situation of the traffic signal to a display unit.
 5. The recognition device according to claim 1, wherein the at least one processor is configured to determine the lighting situation of the traffic signal using the predetermined time, with respect to color display of the traffic signal that indicates whether traveling of the moving object is permitted.
 6. The recognition device according to claim 5, wherein in a case of determining a lighting situation of arrow display of the traffic signal indicating a travelable direction of the moving object, the at least one processor is configured to use, instead of the predetermined time, a second predetermined time longer than the predetermined time.
 7. The recognition device according to claim 1, further comprising: a communication unit that receives information related to lighting of the traffic signal by wireless communication, wherein even in a case where a time during which the same lighting mode of the traffic signal is continuously detected is less than the predetermined time, the at least one processor is configured to determine the lighting situation of the traffic signal based on the information in a case where the communication unit receives the information.
 8. The recognition device according to claim 7, wherein the communication unit receives, from an external communication apparatus installed on a road on which the moving object travels, the information regarding a plurality of traffic signals present around the moving object, and the at least one processor is configured to: determine a lighting situation based on the information, regarding a traffic signal of which an elapsed time from reception of the information by the communication unit is within a prescribed period among the plurality of traffic signals, and determine the lighting situation based on a result of detecting the lighting mode, regarding a traffic signal of which the elapsed time is not within the prescribed period among the plurality of traffic signals.
 9. The recognition device according to claim 1, wherein in a case where a plurality of traffic signals are included in an image obtained by the imaging unit, the at least one processor is configured to obtain a degree of certainty for each of the traffic signals, the degree of certainty indicating certainty that the traffic signal is a traffic signal in the traveling direction of the moving object, and detects a lighting mode of a traffic signal of which the degree of certainty is highest among the plurality of traffic signals.
 10. The recognition device according to claim 1, wherein in a case where a determination result of the lighting situation of the traffic signal is used for automatic control of steering and a vehicle speed of the moving object, the at least one processor is configured to set the predetermined time longer than that when the determination result is not used for the automatic control.
 11. A moving object comprising the recognition device according to claim
 1. 12. A recognition method for recognizing a lighting situation of a traffic signal, the recognition method comprising: detecting sequentially, for each image periodically obtained by an imaging unit that periodically images an external environment of a moving object, a lighting mode of the traffic signal included in the image; and determining, in a case where a same lighting mode of the traffic signal is continuously detected for a predetermined time in the detecting, the lighting mode as a lighting situation of the traffic signal, wherein the predetermined time is twice or more as long as an imaging cycle of the imaging unit.
 13. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the recognition method according to claim
 12. 